Learning Disjunction of Conjunctions

The quest ion of whether concepts expressible as d is junc t ions of conjunct ions can be learned from examples in polynomial t ime is i nves t i ga ted . Pos i t i ve resu l t s are shown fo r s i g n i f i c a n t sub­ classes tha t a l low not only p ropos i t i ona l p re ­ d icates but also some r e l a t i o n s . The algori thms are extended so as to be provably t o l e ran t to a ce r ta i n q u a n t i f i a b l e e r ro r ra te in the examples data. I t i s f u r t he r shown tha t under ce r ta in r e ­ s t r i c t i o n s on these subclasses the learn ing a lgo­ r i thms are w e l l su i ted to implementation on neural networks of threshold elements. The possib le im­ portance of d is junc t ions of conjunct ions as a know­ ledge representat ion stems from the observations tha t on the one hand humans appear to l i k e using i t , and, on the o ther , tha t there is c i rcumstan t ia l evidence tha t s i g n i f i c a n t l y la rger classes may not be learnable in polynomial t ime. An NP-completeness r e s u l t corroborat ing the l a t t e r is a lso p re ­ sented. 1 . In t roduc t ion The a b i l i t y of humans to learn new concepts is remarkable and myster ious. L i t t l e progress has been made in the problem of s imula t ing t h i s process by computer. Worse s t i l l there is l i t t l e agreement on what r e a l i s t i c spec i f i ca t ions of the desired behavior of such a s imula t ion might look l i k e . The approach taken to t h i s problem in a previous paper [13] was to appeal to computational complexity theory to provide some guidel ines on t h i s l a s t quest ion . In p a r t i c u l a r , var ious classes of prop­ o s i t i o n a l concepts were explored tha t could be learned from examples, or sometimes o rac les , in a polynomial number of computational s teps. Pre­ sumably b i o l o g i c a l systems cannot learn classes tha t are computat ional ly i n t r a c t a b l e . The o v e r a l l model was h i e r a r c h i c a l . A concept to be learned was expressed as a p ropos i t i ona l expression in terms of concepts already known. I f t h i s expression was too complex it was computation­ a l l y i n f eas i b l e to learn i t . The maximal complex­ i t y of such expressions tha t could be learned feas ib l y was s tud ied . That i nves t i ga t i on pointed to one p a r t i c u l a r concept class tha t espec ia l l y warranted fu r t he r i nves t iga t ion those tha t could be expressed as a ♦Supported in pa r t by Nat ional Science Foundation grant MCS-83-02385. d i s j unc t i on o f conjunct ions, ca l l ed d i s j unc t i ve normal form (DNF). The a t t r a c t i o n of t h i s class is tha t humans appear to l i k e i t f o r represent ing knowledge as is evidenced, f o r example, by the success of the product ion system paradigm [6] and of Horn clause log ics [ 8 ] . Our inves t iga t ions sug­ gest tha t s i g n i f i c a n t l y larger classes may not be learnable in polynomial t ime. Hence t h i s class may tu rn out to have a cen t ra l r o l e . This paper is organized as f o l l ows . In Sec­ t i o n 2 the formal framework w i l l be described f i r s t . I t w i l l then be shown tha t s i g n i f i c a n t sub­ classes of p ropos i t i ona l DNF expressions can be learned in polynomial time j u s t from examples. Whether the whole class can be so learned remains an open problem. The sect ion fo l low ing tha t w i l l discuss how the above resu l t s can be extended to al low v a r i ­ ables and re l a t i ons in the manner of predicate ca lcu lus . The extension is to a very r e s t r i c t e d par t of the predicate calculus where there is only e x i s t e n t i a l q u a n t i f i c a t i o n and a l i m i t e d number of va r i ab les . In Section 4 the question as to whether the above a lgor i thm can be made robust to er rors in the data is discussed. I t is shown tha t the learn ing algori thms can indeed be made robust to a small ra te of even mal ic ious ly constructed e r ro r s . The s u i t a b i l i t y of our learn ing algori thms to implementations on d i s t r i b u t e d " b r a i n l i k e " models of computation is the subject of Section 5. In the p ropos i t i ona l case, under ce r ta in na tu ra l r e s t r i c ­ t i o n s , such implementations are immediate, but t h i s does not appear to extend to the case al lowing r e ­ l a t i o n s . F i n a l l y in Section 6 we observe tha t the l i m i t s o f l e a r n a b i l i t y are not fa r o f f . Even i f i t i s known tha t a s ing le pure ly conjunct ive expression explains f i f t y percent of the occurrences of a con­ cept , d iscover ing or approximating t h i s conjunct ion may be NP-hard. The word learn ing has been used in numerous senses in phi losophy, psychology and A I . A usefu l d i s t i n c t i o n to make in t h i s context is whether the ava i lab le data from which the learn ing or induc t ion is to be made is consistent w i t h many var ied hypoth­ es i s , or essen t i a l l y w i t h j u s t one. In the former s i t u a t i o n , which i s o f ten t a c i t l y accepted in the philosophy of science, and also in machine learn ing